Goto

Collaborating Authors

 non-normal matrix


Reviews: Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

Neural Information Processing Systems

This paper proposes nnRNN, which is the non-normal matrix relaxation to improve the expressivity of RNNs with unit eigenvalue constraints. Previous studies use the unitary or orthogonal matrix constraints to prevent the exploding and vanishing gradient problems in RNNs, but these constraints deteriorate the performance in empirical tasks due to the strong constraints. This paper relaxes the unitary matrix (which is a normal matrix that has eigenvalues on the unit circle) constraints as allowing the non-normal matrices that have the eigenvalues on a unit circle. In addition, this paper theoretically and empirically shows how the non-normal relaxation affects the dynamics of RNNs. This paper is well written, and the motivation of non-normality is explained thoroughly.